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ABSTRACT 

In this paper, we advance that there are several issues pertaining to the design of research in 
instructional technology. It is our view that much of the current research taking place may suffer 
from poor quality, inappropriate design, and lack of social responsibility. We contend that the 
most appropriate way to research the effectiveness of online learning is the use of design 
experiments. We present an exemplar of a recent design experiment that was completed at a 
university in Johannesburg, South Africa. During this study, the researchers explored the extent 
to which complex thinking skills can be facilitated in online learning environments. A design 
experiment was engineered in which a learning programme was designed and developed for 
Masters students. Specific instructional methodologies were employed in the learning 
programme, and activities were designed that facilitate the use of complex thinking skills. The 
extent to which these skills were evident in student online activities was easily detected by using 
the comprehensive checklists and rubrics that were generated. A rigorous framework for analysis 
was developed. The findings were integrated with theoretical perspectives on instructional 
strategies for complex thinking development and new, unique criteria for online learning design 
were yielded. We are of the view that the findings of our study are ‘true’, as the appropriate 
methodology was used to conduct it. 

Keywords: Design experiments, complex thinking, instructional methodologies, online learning, 
unigue criteria. 


INTRODUCTION: QUESTIONING INSTRUCTIONAL TECHNOLOGY RESEARCH 

The decision whether to use some form of instructional technology in education should be based 
on the question: Is the use of instructional technology likely to improve education? (Mitchell 1997; 
Reeves 1995). The way in which scholars, lecturers or teachers attempt to establish whether 
such interventions are indeed beneficial is through a process of scientific research in the form of 
case studies, course evaluations or experimental studies. In fact, the literature abounds with 
reports in which the benefits of instructional technology interventions are espoused. Lockard and 
Abrams (2001) list many research studies in which it has been found that the use of instructional 
technology shows gains in subject-matter achievement, learning retention and speed, attitudes 
towards learning, problem solving and for students who are at risk. We assert in this paper that 
the research results pertaining to instructional technology research may be flawed due to poor 
quality research and inappropriate research designs. We further assert that an academic system 
that rewards research that is not socially responsible will not produce relevant and high quality 
research. We will argue that design experiments (development research) that are executed 
rigorously will address the concerns that we have about instructional technology research. 

There is significant evidence that the research results pertaining to the benefits of using 
interactive technologies to support teaching and learning is questionable, often because of a lack 
of rigour during the execution of the research. According to Reeves (2000), the “quality of 
published research in Instructional Technology is generally poor”. Reeves (1995) launches a 
scathing attack on research done in instructional technology, and claims that most published 
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research articles are “pseudoscience” (also see Mitchell 1997) He claims, after an analysis of five 
articles published in refereed journals, that these articles have specification errors, have few links 
to robust theory, have inadequate literature reviews and treatment implementation, have 
measurement flaws and inconsequential outcome measures, inadequate sample sizes, 
inappropriate statistical analyses and meaningless discussion of results. Dillon and Gabbard 
(1998), who reviewed 500 papers for an article they prepared for the journal, Review of 
Educational Research, found that only 30 of these met the minimal criteria for good scientific 
studies for inclusion in their review. Reeves, Mitchell and Stokes are not the only dissenting 
voices in the research community who have expressed concern about the state of instructional 
technology research. In fact, Reeves (1995) refer to authors like Mielke (1968), Lumsdaine 
(1963), Schramm (1977), Clark (1983) and Salomon (1991), who were the forerunners in the 
questioning of research practice in instructional technology. In his seminal work, Clark (1983, 
1994) asserts that media (and therefore instructional technology) has no influence on learning 
and he criticises the research in this field. He explains that meta-analytic reviews report an 
approximate 20% increase in evaluation scores following the use of instructional technology in 
comparison to conventional forms of teaching. However, he contends that it is the instructional 
methodology that underpins these interventions that account for the gains in learning of those 
research reports. The research studies that have examined the use and effectiveness of the 
media used therefore failed to isolate the real reasons for the learning gains that were 
demonstrated. The publication of Clark’s initial work sparked the well-reported Clark-Kozma 
debate, wherein the two opposing sides drew the proverbial line in the sand about the value of 
media (instructional technology) for learning. A primary thrust in this debate was the selection of 
appropriate methodologies for researching instructional technology. 

The root of the problem may possibly be found in the ‘quantitative-qualitative’ paradigm debate. 
Hoepfl (1997) explains that the relative value of qualitative or quantitative inquiries has been 
raging for a long time. Quantitative research is based on an experimental design in which a 
hypothesis is tested and from which generalisations can be drawn. Reeves and Hedberg (2003) 
describe this type of research as “analytic-empirical-positivist-quantitative”. Many researchers 
claim that positivist, experimental designs are the only appropriate ones for doing valid and 
reliable research. In fact, Reeves (2000) found that most published research in leading journals 
for education was situated within the quantitative, positivist paradigm. Qualitative research on the 
other hand does not rely on numerical or statistical data and attempts to understand phenomena 
in “natural settings” (Hoepfl 1997). Strauss and Corbin (1990, p.17) define qualitative research as 
producing results that are not “arrived at by means of statistical procedures or other means of 
quantification”. Many scholars are of the opinion that research in education should be based on 
qualitative data. In addition to these two paradigms, Soltis (1992) explains that research can be 
situated within a ‘critical theory’ paradigm. Critical research aims to critique the social order to 
bring about change and examines restrictive and alienating conditions. It questions the 
maintenance of the status quo and wants to bring about cultural, political and social change. 

The question is which of these paradigms (or combination of paradigms) is suitable for 
researching instructional technology. Roblyer and Knezek (2003) claim that research findings that 
confirm the benefits of modern technologies for learning may “simply not hold true” as much of 
such research was done using behaviourist-cognitivist approaches to assessing learning benefits. 
Alternatively, in these research projects, comparisons between technology-mediated learning 
environments and traditional face-to-face course deliveries using experimental or quasi- 
experimental methodologies were made. Some researchers like Tellez (1993), Hoepfl (1997) and 
Reeves (1995, 2000) claim that it is not possible to conduct true experimental designs in social 
science inquiries. Because of the fact that researchers are often faced with intact groups (specific 
classes or groups) that cannot be divided up for random assignment and the creation of 
experimental and control groups, true experimental designs are often simply not viable. In this 
regard, the question further needs to be asked what the aim of a research project is. Reeves and 
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Hedberg (2003) point out that the reliance of experimental methodologies stem from the need to 
“prove” the effectiveness of a particular educational intervention, in other words, the research has 
a summative evaluation dimension. Many of these research projects are case studies. Case 
studies appear to exemplify the “Tylerian Objectives-Based Evaluation Model”, which would judge 
a programme to be good if the set objectives were achieved (Reeves & Hedberg 2003). Case 
studies appear to be underpinned by ‘after-the-fact’ methodologies, and may seem wasteful if 
some contribution to theory is not made. Suitably engineered educational online interventions that 
are meticulously designed and that are situated within specific educational theory, may therefore 
be of more value to learners. Additionally, when the impact or effectiveness of such interventions 
is scrutinised and researched, appropriate methodologies need to be utilised that go beyond the 
mere exploration of cases. Cunningham (Willis 1994) claims that it is impossible to produce 
‘findings’ that are generalisable across all possible circumstances, and specifically so within 
social science contexts. Constructed knowledge is not ‘truth’ that remains stable and dependable 
forever, rather, it exists within specific contexts and perspectives - knowledge that may profess to 
be truth for one context may very well not be ‘truth’ for other contexts. Therefore, we advance that 
empiricist designs that depend on pre-testing and post-testing using quantitative data may not be 
the most appropriate way of researching online learning. Subsequently, we hold the view that 
research design in social science can at best be quasi-experimental designs. 

The third dimension that impacts on the quality of instructional technology research is the way by 
which scholars are rewarded for their research outputs. Reeves (2000) describes in a paper 
delivered at the prestigious America Educational Research Association (AERA) his experiences 
when appointed at a university as a junior professor. He explains how he was told to collect “lots 
of data” in order to publish and therefore advance in the university system. He points out that the 
state in which his appointment was made had a documented poor educational system, but he 
was not told to find solutions, through research, for those problems. This exemplifies the ‘publish 
or perish’ notion, which is a significant challenge that faces higher education. It is our experience 
in the higher education system that academics are under pressure to publish (do research). 
Publishing is incentivised by the higher education institutions, which receive financial rewards in 
the form of subsidy, and which in turn reward academics with promotion. Whereas we do not 
question the reward system for research, we would plead for a system wherein ‘socially 
responsible’ research is advanced. In this regard, Reeves (1995) refers to ‘socially responsible’ 
research as research that aims to make education better, therefore finding practical solutions to 
real problems. It highlights the fact that much educational research may have little value for 
solving the practical problems that plague education in general (also see Reeves 1995, 2000). 
Similarly, Stokes, (1997) in his Pasteur’s Quadrant: Basic Science and Technological Innovation 
work explains that much of the research done in the educational field contributes little to the 
understanding of the theories that underpin education (and in our view also that of instructional 
technology) and that these studies do not advance fundamental knowledge in the relevant 
knowledge domains. He uses as exemplar the work done by Louis Pasteur, who found practical 
solutions for real-world problems and at the same time advanced fundamental (theoretical) 
knowledge, in this case about the preservation of fresh food. We acknowledge that our view may 
be contentious. Reeves (1995) points out that others in the research community will argue that 
the search for the sake of knowledge’s sake is paramount, and that researchers should not be 
prescribed to as to what they should research. Although we concur that a purist agenda is 
important for the maintenance of independent scholarship, we would like to have - in the context 
of the problems that were highlighted with regard to instructional technology research, and to the 
further context of the educational problems that beseech South Africa in general - a research 
agenda developed that advances both theory and practical application. In this paper we argue 
that design experiments will address these dual needs. 

The South African situation is unclear. The most typical application of qualitative research in 
instructional technology seems to be that of case studies. Van der Westhuizen (2002) conducted 
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a meta-analysis of research topics and methodologies in South Africa that related to instructional 
technology research. He found that the vast majority of published research is case studies. 
Although the value of case studies in a developing field of knowledge is not to be underestimated, 
we doubt that this approach will lead to fundamental understanding of the theories that are 
associated with online learning. Although they may highlight practical problems, and even 
suggest solutions to those problems, the findings need to be incorporated into existing theory. 
Whether case studies yield sufficient in-depth data to advance fundamental knowledge remains 
to be seen. No other meta-studies that have examined the research designs of instructional 
technology inquiries have been found in South African literature. 


DESIGN EXPERIMENTS 

In this paper, we contend that the most appropriate way to research the effectiveness of online 
learning is the use of design experiments. We assert that design experiments address the 
concerns that we have raised in the previous section. In the first place, we argue that design 
experiments require rigorous designs that yield rich, in-depth data over a prolonged period of 
time, and therefore by virtue of the design addresses issues of quality, depth and validity. 
Secondly, design experiments may use any of the paradigms that underpin educational research, 
and in fact, will utilise both approaches in a complementary manner. Thirdly, as design 
experiments address real-life problems and attempt to engineer solutions to those, we believe 
that design experiment methodologies are socially responsible. The following section provides a 
definition of the concept, and outlines the goals of design experiments. 

Conceptualisation 

The term “design experiments” - also referred to as “formative experiments” (Barab & Kirshner 
2001), “applied research” (van den Akker 1999; Reeves 2000), “use-inspired basic research” 
(Stokes 1997) or “development research” (Reeves 2000) - was introduced in 1992 by Brown and 
Collins. More recently the term “design research” has been applied to this kind of research (Barab 
& Kirshner 2001 and Collins 1999). The terms “design experiments” and “design research” will be 
used interchangeably in this paper. Design experiments are types of research that place 
educational experiments in real-world settings to find out what works in practice (RooseveltHaas 
2001). According to Cobb et al. (2003), design experiments entail both “engineering” particular 
forms of learning, and systematically studying those forms of learning within the context defined 
by means of supporting them. This designed context is subject to test and revision, and the 
successive iterations are similar to systematic variation in experience. Design experiments 
incorporate the notion of formative and summative evaluation of learner skills and knowledge 
demonstrated over time, penetrating into the learning processes on a weekly schedule, as 
instructors and researchers negotiate instructional decisions (Brown 1992). Design experiments 
are pragmatic as well as theoretical in orientation in that the study of function - both of the design 
and of the resulting ecology of learning - is at the heart of the methodology (Cobb et al. 2003). A 
design science in education therefore aims at determining how the design of learning 
environments contributes to learning (Brown 1997). 

The goals of design research 

Design experiments were developed as a way of conducting formative research for testing and 
refining educational problems, solutions and methods (Reeves 2000; Stigler & Hiebert 1999). 
They are mainly used by researchers with development goals in mind (Reeves 2000). The goals 
of design experiments (development research) as described by Reeves are summarised in Figure 
1 . 
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Design experiments 



Refinement of problems, solutions and methods 


Figure 1: Development approach to research (Reeves 2000) 


However, design research is not aimed simply at refining practice. It should always have the dual 
goal of refining both theory and practice (Edelson 2001; Joseph 2000). Design experiments are 
conducted for the generation and testing of theories that target domain-specific learning 
processes (Cobb et al. 2003). It ideally results in greater understanding of a learning ecology - a 
complex, interacting system involving multiple elements of different types and levels - by 
designing its elements and by anticipating how these elements function together to support 
learning (van den Akker 1999; Brown 1997; Cobb et al. 2003; Reeves 2000). Design 
experiments, therefore, constitute a means of addressing the complexity that is the hallmark of 
educational settings (Barab & Kirchner 2001). Elements of a learning ecology typically include the 
tasks or problems that learners are asked to solve, the kinds of discourse that are encouraged, 
the norms of participation that are established, the tools and related material means provided, 
and the practical means by which instructors can orchestrate relations among these elements 
(Cobb et al. 2003). 

The researcher firstly develops the broader theoretical goals of the study (a design focus), frames 
selected aspects of the envisioned learning (provides a theoretical framework for the study), 
specifies the settings in which the learning will take place as well as the means of supporting it, 
and develops a model of the learning tasks and instructional strategies that can support that 
learning (Brown & Campione 1996). The process of engineering or specifying the forms of 
learning being studied provides the researcher with a measure of control not obtainable in purely 
naturalistic investigations. 

Design experiments, according to Cobb et al. (2003), have two faces: prospective and reflective. 
On the prospective side, designs are implemented with a hypothesised learning process and the 
means of supporting it in mind, in order to expose the details of that process to scrutiny. An 
equally important objective is to foster the emergence of other potential pathways for learning and 
development by capitalising on contingencies that arise as the design unfolds. The theory 
therefore informs the design focus and prospective design (DiSessa 1991). On the reflective side 
design experiments are conjecture-driven tests, assessing the critical design elements, often at 
several levels of analysis (Shepard 2000). Together the prospective and reflective aspects of 
design experiments result in an iterative design process featuring cycles of invention and revision 
(Cobb et al. 2003). The evaluation of the design, therefore, is an ongoing process that changes 
as the design changes (Brown & Campione 1996). 


RESEARCH EXAMPLE: COMPLEX THINKING ONLINE 

We provide as exemplar a recent design experiment which was completed at a university in 
Johannesburg, South Africa. During this study, the researchers explored the extent to which 
complex thinking skills could be facilitated in online learning environments. In this study, a one- 















110 IJEDICT 


on-one design experiment with a small number of learners was engineered. A learning 
programme was designed and developed for Masters students who were enrolled for a course in 
Instructional Technology. The aim was to create a small-scale version of a learning ecology for 
in-depth and detailed study (Barab & Kirshner 2001; Cobb et al. 2003) and to refine the design 
parameters for a new type of curriculum. The research suggested in this study looked at a 
complex system of interrelated factors and events, where each component, event or action has 
the potential of affecting the unit as a whole (Collins 1999). There is compatibility in this research 
between the systemic nature of the subject matter and the use of qualitative research methods. 
The research methodology for this study was guided by principles of interpretive inquiry outlined 
by researchers such as Lecompton, Preissle and Renate (1993) and Miles and Huberman (1994). 
The research was conducted in four phases as summarised in Figure 2. 


Design Experiment 



Figure 2: Using a design experiment for assessing a learning programme for complex 
thinking development 


Phase A: Establishing a theoretical framework for the study 

The development of the qualitative/interpretive design experiment began with the establishment 
of a theoretical framework, the set of questions to be answered by the research. The framework 
address the problem to be investigated by the study, reviewing what is known about the topic, 
what is not known, why it is important to know it, and the specific purpose of the study 
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(Winegardner 2000). Merriam (1992) stresses the importance of identifying the theoretical 
framework that forms the ‘scaffolding’ or underlying structure of the study. Theory should be 
present in all qualitative studies because no study could be designed without some question 
being asked explicitly or implicitly. The phrasing of that question and the development of a 
problem statement reflect a theoretical orientation (Merriam 1992). The literature study, therefore, 
formed a theoretical and analytical framework of criteria, serving as a foundation for the analysis 
and interpretation of the data collected during the research project, and, this, according to Vockell 
and Asher (1995), directs the questions asked by the researcher. It also helps the researcher 
identify methodological techniques used to research similar phenomena as well as contradictory 
findings. The aim of the literature review in this study was to identify the following: criteria for the 
development of complex thinking, instructional strategies that could enhance complex thinking 
development, and methods of using online learning for the advancement of complex thinking 
development in a Web-based learning environment. Course content was then designed according 
to these findings and presented in the Web-based learning environment. 

The following objectives were realised in Phase A: 

Objective 1 

Researched the essential characteristics of complex thinking through a literature study and 
derived criteria for identifying complex thinking. 

Objective 2 

Through a literature study, possible instructional strategies and techniques to enhance complex 
thinking were thoroughly researched and a set of criteria derived. 

Objective 3 

Through a literature study the contribution of Web-based learning to the learning process was 
researched and a set of criteria derived. 

The elements (criteria) identified in this phase of the study provided a framework for the design of 
the Web-based learning programme developed in Phases B and C of the study. 

Phase B: The design and development of the Web-based learning programme 

In Phase B of the study, a learning environment was designed to incorporate the criteria 
established in Phase A of the research. During this stage the critical elements of the design and 
their relevance to each other were identified. The design included a contact session, serving as 
an introduction to the theme. The second part of the design experiment comprised a series of 
Web-based learning activities, which incorporated various instructional methodologies to 
facilitate/enhance complex thinking. Different discussion forums were created in the Web-based 
learning environment to facilitate these activities. The programme was implemented in Phase C 
of the study. 

Phase C: The implementation of the Web-based learning programme 

During Phase C of the inquiry, the Web-based programme was implemented using a series of 
instructional strategies focussing specifically on complex thinking. Specifically, Phase C sought to 
answer the following questions: 
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• What types of complex thinking skills did learners employ while interacting in the Web-based 
learning environment? 

• How did the instructional strategies and techniques employed in the Web-based learning 
environment impact on the facilitation of complex thinking? 

• How did the Web-based learning activities contribute to the success of the course? 

• The extent to which these complex thinking skills were evident in the student online activities 
could therefore easily be detected by using the comprehensive checklist and the criteria that 
were generated. 

Phase D: Data analysis 

Phase D provided an explicit account or report of the outcomes of the research, according to the 
criteria specified in Phase A, and types of evidence used. Data were collected from submissions 
and discussions in the Web-based learning environment and these were interpreted against 
theoretical criteria derived from the literature study. The data that were collected were reduced to 
several themes (complex thinking, instructional strategies and Web-based learning) with several 
categories and sub-categories of criteria, and provided a framework for the analysis and 
interpretation of the data by using a classification scheme One of the most important tasks of 
analysis is the identification of “patterns, commonalities, differences and processes” (Miles & 
Huberman 1994). Categories (criteria) were developed in terms of their properties and some 
categories were eventually promoted to major categories while others were demoted to sub¬ 
categories. 

A practical format for the analysis of the written discussions (talk) and assignment activities 
(described as ‘messages’ by the Web-based software WebCT used to facilitate the learning) 
displayed in the Web-based learning environment had to be found. In this study content analysis 
was regarded as the most useful model for analysing the content of these recorded messages in 
accordance with Merriam’s (1992) emphasis on the importance of observing and analysing the 
content of learners’ conversations. The learners’ discussions were divided into units of meaning 
as the most practical method for this study. This method counts each type of talk as it occurs 
(Henri 1992). 


SUMMARY OF FINDINGS 

The findings were integrated with theoretical perspectives on instructional strategies for complex 
thinking development and, new, unique criteria for online learning design were yielded. This 
research is not generalisable, and instructional practitioners, designers and learners will have to 
judge the applicability of the findings and recommendations made. 

There are many implications for practice in the findings of this research. Most relate directly to the 
use of Web-based learning in higher education environments, although many will apply to other 
classroom settings. The implications pertain to both the design of online learning and the 
application of instructional strategies used in instructional designs. The contribution of this 
research is three-fold. It is significant in the South African context, it has practical value and 
design criteria for Web-based learning were generated and documented to produce design 
principes that may be useful to any practitioner of Web-based learning. 

Significance in the South African context 

The major contribution of this study is that, for the first time in the South African context, research 
was undertaken based on a typology that clustered the dimensions of complex thinking, 
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instructional strategies/techniques and Web-based learning within the context of a design 
experiment. This research is significant for higher education in South Africa where Web-based 
learning is emerging as a tool to facilitate instruction. Prime reasons for using Web-based 
learning in South Africa are to improve the quality of learning, to provide learners with everyday 
information technology skills they will need in their career and personal lives, and to widen access 
to education and training. As Web-based learning is being implemented, an important emergent 
issue is to ensure that learning is adequately supported and facilitated. This study aimed at 
generating criteria to support meaningful learning in a Web-based learning environment and 
criteria were generated for providing clear learning outcomes, engaging learners, and structuring 
learner interactions to facilitate thinking development. In South Africa there is a need for the 
development of thinking skills as a general thrust in education and this research is particularly 
relevant as the development of critical and creative thinking skills (complex thinking skills) has 
been identified as a national critical outcome. 

Practical value 

Furthermore, this study has practical value because criteria were applied to a practical Web- 
based learning environment. This study focused on enhancing the practice of Web-based 
learning by linking the practical to the theoretical foundations and adequate literature reviews 
This research therefore aimed at making both a practical and scientific contribution to ensure a 
more productive inquiry. Furthermore, there were sufficient theoretical principles to guide the 
practice (Reeves 2000). The researcher aimed at explaining the phenomenon of complex thinking 
development through the logical analysis of learning theories and Web-based learning principles. 
However, because there are no sacred steps to effective instruction, this research - focusing on 
how Web-based instruction works - tested conclusions related to the theories of teaching, 
learning, thinking, assessment, social interaction, instructional design, and so forth. In addition, 
the primary goal of this design experiment was the development of a profile rather than testing 
hypotheses (Collins 1999). The overall goal of this research was therefore to solve real world 
problems while at the same time constructing design principles that can inform future designs 
(Reeves 2000). With this research goal in mind, it was considered necessary to employ a design 
experiment as research method. 

The implications for the selection of instructional strategies 

This design experiment aimed to determine the effects of Web-based instructional strategies on 
complex thinking development under certain controlled conditions. The principal implication for 
instructional designers is that the quality of the learning that takes place (whether in the Web- 
based learning environment or normal classroom settings) is directly influenced by the 
instructional strategies used. There are many advantages to be gained from implementing 
instructional strategies in a manner that supports the construction of knowledge and enhances 
complex thinking development. 

A major implication for instructors and learners is that, contrary to constructivist beliefs, direct 
instruction plays a vital part in ensuring the quality of learning and thinking. If basic skills are not 
taught, learners will not be able to understand and apply these on higher levels of thinking. 
Learners should, for example, be taught how to apply the action words that describe the 
outcomes; they need to be taught the skills of co-operative learning and need knowledge on a 
topic to be assessed, in order to complete such complex tasks as peer assessment and group 
work, particularly in a complex learning environment such as the Web. Second-language users 
often find it difficult to understand the outcomes and assessment questions posed to them and 
the instructor should ensure that these are explained properly. 

An important finding of the research is that the action words that describe the learning outcomes 
should be derived from the different complex thinking skills sets, because the outcomes 
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employed directly affect the degree of complex thinking that takes place. It also found that time 
frames should not place restrictions on learning activities as it takes time to learn and think. Time 
frames should therefore be flexible and realistic to allow learners reasonable time to complete 
learning activities and work at their own pace. Furthermore, the research finds that co-operative 
learning strategies can be advantageous, but there are also some disadvantages. In particular, it 
suggested that inexperienced members should first be taught the basic skills of co-operative 
learning, and the instructor should ensure that these activities are clearly defined and procedures 
specified. Working in groups was found to take up much more time than working alone, therefore 
time should be given to complete group activities, especially in Web-based asynchronous 
environments where interaction is delayed. 

An additional finding is that the instructor should apply questions that focus on the higher levels of 
cognitive activity (ill-structured questions) throughout the learning process to direct the 
discussions and to stimulate the learners’ thinking. Web-based learning activities should be 
monitored and assessed regularly to ensure that learners are provided with the necessary 
feedback, motivation and guidance. This will also help the instructor to intervene and alter the 
learning, if and where necessary. 

The implications for the design of online learning programmes 

The principal implication for instructors is that instructional design models for Web-based learning 
can be an effective substitute for the traditional classroom design model. Contrary to concerns 
that Web-based learning models may place the focus on instruction and not on learning, an 
environment was created where learners actively used complex thinking skills in collaborative 
group settings. The research indicated that, generally, Web-based learning strategies could be 
successfully used for the facilitation of complex thinking. The seven Web-based learning criteria 
that were generated, may guide designers of Web-based instructional designs to a model based 
on outcomes-based education principles and learning theory. 

A major implication for current research is that some learners may find it very difficult to adapt to 
new didactic methods, such as problem-solving activities and group work (peer assessment and 
debate). If, in this situation, they are also required to apply additional skills such as using the 
Web-based discussion forums effectively, the instructor must ensure that these skills are taught in 
advance and that the learners are familiarised with the specific Web-based learning settings, 
before an attempt is made to let them participate in such a complex activity. The new instructional 
strategies and techniques employed in the Web-based learning environment are geared to self- 
direction and active participation and some learners take time to adapt to these new approaches. 
An important implication for learners and instructors is that the Web as medium for instruction 
should be carefully weighed to ensure that flexible learning is provided. Time settings should be 
flexible, and adequate time should be given to complete group activities, especially in 
asynchronous Web-based learning environments where the interaction is delayed. Without some 
time constraints however, assignments are not completed and marked in time, and proper 
feedback is not provided. 

Design principles for Web-based learning 

This study provides a framework incorporating design principles for instructors and designers of 
Web-based learning environments to encourage/faciltate complex thinking. This framework 
includes: 

• Criteria for identifying complex thinking and providing learning opportunities where the learner 
is encouraged to demonstrate and develop specific abilities and skills in complex thinking; 

• Instructional criteria/requirements for the effective facilitation of complex thinking, as derived 
from the social and cognitive constructivist learning theories; 
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• Instructional design criteria applicable to asynchronous Web-based learning environments for 
the facilitation of complex thinking and effective learning. 

The thorough exploration of the three theoretical thrusts of this study (complex thinking, 
instructional strategies/techniques and Web-based learning) makes a significant contribution and 
the list of criteria developed is potentially of great value to other researchers, instructors and 
practitioners of Web-based learning. 


CONCLUSION 

In this paper our aims were three-fold. In the first place, we wanted to highlight some of the 
issues pertaining to instructional technology research. We concluded that a number of factors 
impacted on past instructional technology research, being poor quality research, problems 
associated with research designs and research that is not socially relevant. We then proposed 
that design experiment methodologies may address many of the concerns that we have 
identified. The design experiment is a particulary suitable strategy to research implementations in 
educational hypermedia, but this methodology is under-utilised in the South African context. 
Finally, we constructed, as exemplar, our own design experiment. The paper described a 
framework for the design of such an experiment in which the development of complex thinking 
skills in Web-based learning environments were envisaged. The meticulous application of design 
experiment methodology illustrated the appropriateness of this strategy for the research of 
instructional technology. 
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